Skip to content

Conversation

@abhishekdas99
Copy link
Contributor

@abhishekdas99 abhishekdas99 commented May 12, 2020

HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for branch
+1 💚 mvninstall 19m 54s trunk passed
+1 💚 compile 19m 17s trunk passed
+1 💚 checkstyle 2m 40s trunk passed
+1 💚 mvnsite 3m 1s trunk passed
+1 💚 shadedclient 20m 17s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 7s trunk passed
+0 🆗 spotbugs 3m 3s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 6s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 24s Maven dependency ordering for patch
+1 💚 mvninstall 1m 56s the patch passed
+1 💚 compile 16m 24s the patch passed
+1 💚 javac 16m 24s the patch passed
-0 ⚠️ checkstyle 2m 38s root: The patch generated 10 new + 59 unchanged - 0 fixed = 69 total (was 59)
+1 💚 mvnsite 2m 59s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 5s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 6s the patch passed
+1 💚 findbugs 5m 33s the patch passed
_ Other Tests _
-1 ❌ unit 9m 23s hadoop-common in the patch passed.
-1 ❌ unit 95m 42s hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 1s The patch does not generate ASF License warnings.
224m 14s
Reason Tests
Failed junit tests hadoop.ha.TestZKFailoverController
hadoop.fs.ftp.TestFTPFileSystem
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4d3351895c4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / bd342be
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/testReport/
Max. process+thread count 4804 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/1/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for branch
+1 💚 mvninstall 19m 10s trunk passed
+1 💚 compile 16m 59s trunk passed
+1 💚 checkstyle 2m 36s trunk passed
+1 💚 mvnsite 2m 58s trunk passed
+1 💚 shadedclient 19m 59s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 3s trunk passed
+0 🆗 spotbugs 3m 1s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 5s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 1m 55s the patch passed
+1 💚 compile 16m 56s the patch passed
+1 💚 javac 16m 56s the patch passed
-0 ⚠️ checkstyle 2m 34s root: The patch generated 1 new + 59 unchanged - 0 fixed = 60 total (was 59)
+1 💚 mvnsite 2m 54s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 15s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 3s the patch passed
+1 💚 findbugs 5m 27s the patch passed
_ Other Tests _
-1 ❌ unit 9m 22s hadoop-common in the patch passed.
-1 ❌ unit 94m 41s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 59s The patch does not generate ASF License warnings.
219m 58s
Reason Tests
Failed junit tests hadoop.fs.ftp.TestFTPFileSystem
hadoop.hdfs.web.TestWebHDFS
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 3630e220fe69 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / bd342be
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/testReport/
Max. process+thread count 4166 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/2/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make it private as we don't need to access this out side.
I think we may not need this at all. You can setConf and getConf.
And look at next comments, we can completely avoid if we use targetFileSystem instead of FileSystem.get(...conf)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use theInternalDir.getFallbackLink().targetFileSystem instead of initializing/getting another fs.
Now conf is not need at all and you dont need above uri also.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

theInternalDir.getFallbackLink().targetFileSystem is going to return ChRootedFileSystem which will wrap the path with fullPath(Path p) method. That was causing error. I have used the underlying filesystem object of the ChRootedFileSystem

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you might have issue was because you are passing fallBackUri. ChrootedFS already contains /fallBackDir and you are again passing a uri which contains /fallbackDir. SO it will fail with "FNFE". If you use chrootedfs, you should just pass relative path by cutting chrooteduri. Since we support only at root, it will be simply "/" on chorootedfs.

I have one another concern that we are returning absolute path with fall back in ls. But with regular mountlinks we will return relative to viewfs scheme ex: [viewfs://default/data, viewfs://default/targetRoot, viewfs://default/danglingLink, [viewfs://default/user2, viewfs://default/user, viewfs://default/internalDir, viewfs://default/linkToAFile]
Since fallback also part of viewfs and its actually a child fs, should we return fallback paths also relatively constructed with respective to viewfs:// scheme?
exmaple: your fallback path is hdfs://nn1/fallBackDir and you have a dir under it. That is /user1
So, when you do ls, should it return viewfs://default/user1 ?
if you use this ls result path to create some directory like viewfs://default/user1/test1, it will/should work ( IMO) and it should create at physical location of hdfs://nn1/fallBackDir/user1/test1
Because you did not have any other mount link with /user1, it will fallback to create at fallback link.

But with the current behavior, we are returning hdfs://nn1/fallBackDir/user1 and if you use this path to create a dir under it, it will actually go to regular hdfs instance but not via viewfs. This could be a potentially problem in that case overloadScheme impl. There hdfs can go to ViewFSOverloadScheme and try to create path as hdfs://nn1/fallBackDir/user1. Now fallBackDir/user1 will be attempted to create in viewfsOS env, it checks whether we have any link with fallBackDir. Of course we will not have any regular link. So, it will create at fallback link.
Our fallback link is already hdfs://nn1/fallbackDir and now it will attempt to create /fallbackDir/user1
So, it will result to create hdfs://nn1/fallbackDir/fallbackDir/user1. This seems not correct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree with your thought. Changed the path back to viewfs. Also in the unit test, verified the creation of directory with the returned path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to add the similar changes in ViewFs.java to cover FileContext based fs impl.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added change in ViewFs.java

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 0s Docker mode activated.
-1 ❌ patch 0m 5s #2010 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #2010
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/3/console
versions git=2.17.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 50s Maven dependency ordering for branch
+1 💚 mvninstall 19m 54s trunk passed
+1 💚 compile 18m 8s trunk passed
+1 💚 checkstyle 2m 48s trunk passed
+1 💚 mvnsite 2m 52s trunk passed
+1 💚 shadedclient 20m 46s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 56s trunk passed
+0 🆗 spotbugs 3m 14s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 21s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 28s hadoop-common in the patch failed.
-1 ❌ compile 0m 52s root in the patch failed.
-1 ❌ javac 0m 52s root in the patch failed.
-0 ⚠️ checkstyle 2m 26s root: The patch generated 5 new + 105 unchanged - 0 fixed = 110 total (was 105)
-1 ❌ mvnsite 0m 33s hadoop-common in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ shadedclient 0m 49s patch has errors when building and testing our client artifacts.
+1 💚 javadoc 1m 20s the patch passed
-1 ❌ findbugs 0m 31s hadoop-common in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 30s hadoop-common in the patch failed.
-1 ❌ unit 102m 17s hadoop-hdfs in the patch passed.
-1 ❌ asflicense 0m 36s The patch generated 4 ASF License warnings.
186m 36s
Reason Tests
Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy
hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
hadoop.hdfs.tools.TestDFSAdminWithHA
hadoop.hdfs.TestFileChecksum
hadoop.fs.TestHdfsNativeCodeLoader
hadoop.fs.viewfs.TestViewFileSystemLinkFallback
hadoop.hdfs.TestReadStripedFileWithDNFailure
hadoop.hdfs.TestDecommission
hadoop.hdfs.TestDistributedFileSystemWithECFile
hadoop.hdfs.TestDecommissionWithStriped
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 04f197480b1a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 0918433
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-compile-root.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-compile-root.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/diff-checkstyle-root.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/testReport/
asflicense https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/artifact/out/patch-asflicense-problems.txt
Max. process+thread count 4348 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/4/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 39s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for branch
+1 💚 mvninstall 20m 13s trunk passed
+1 💚 compile 17m 54s trunk passed
+1 💚 checkstyle 2m 39s trunk passed
-1 ❌ mvnsite 0m 36s hadoop-common in trunk failed.
+1 💚 shadedclient 21m 15s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 59s trunk passed
+0 🆗 spotbugs 3m 33s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 6m 3s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 27s Maven dependency ordering for patch
+1 💚 mvninstall 2m 16s the patch passed
+1 💚 compile 20m 24s the patch passed
+1 💚 javac 20m 24s the patch passed
-0 ⚠️ checkstyle 3m 22s root: The patch generated 5 new + 105 unchanged - 0 fixed = 110 total (was 105)
-1 ❌ mvnsite 0m 34s hadoop-common in the patch failed.
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 16m 54s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 1s the patch passed
+1 💚 findbugs 6m 22s the patch passed
_ Other Tests _
-1 ❌ unit 10m 35s hadoop-common in the patch passed.
-1 ❌ unit 98m 5s hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
236m 20s
Reason Tests
Failed junit tests hadoop.ha.TestZKFailoverControllerStress
hadoop.hdfs.TestRestartDFS
hadoop.hdfs.TestMaintenanceState
hadoop.hdfs.TestFileCorruption
hadoop.hdfs.TestRollingUpgrade
hadoop.hdfs.TestErasureCodingPolicies
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
hadoop.hdfs.TestErasureCodingMultipleRacks
hadoop.hdfs.security.token.block.TestBlockToken
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.TestDFSStripedInputStream
hadoop.hdfs.TestStripedFileAppend
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 1c086f293f4b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 0918433
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/diff-checkstyle-root.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/testReport/
Max. process+thread count 4690 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/5/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 57s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 58s Maven dependency ordering for branch
+1 💚 mvninstall 27m 0s trunk passed
+1 💚 compile 24m 42s trunk passed
+1 💚 checkstyle 2m 53s trunk passed
+1 💚 mvnsite 2m 46s trunk passed
+1 💚 shadedclient 21m 25s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 46s trunk passed
+0 🆗 spotbugs 3m 8s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 15s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for patch
+1 💚 mvninstall 1m 59s the patch passed
+1 💚 compile 17m 15s the patch passed
+1 💚 javac 17m 15s the patch passed
-0 ⚠️ checkstyle 2m 53s root: The patch generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105)
+1 💚 mvnsite 2m 45s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 27s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 43s the patch passed
+1 💚 findbugs 5m 31s the patch passed
_ Other Tests _
+1 💚 unit 9m 20s hadoop-common in the patch passed.
-1 ❌ unit 112m 6s hadoop-hdfs in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
257m 6s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
hadoop.hdfs.TestDecommission
hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.datanode.TestBPOfferService
hadoop.hdfs.TestPersistBlocks
hadoop.hdfs.TestRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/6/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 2823e4b2b469 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 0918433
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/6/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/6/testReport/
Max. process+thread count 3195 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/6/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you might have issue was because you are passing fallBackUri. ChrootedFS already contains /fallBackDir and you are again passing a uri which contains /fallbackDir. SO it will fail with "FNFE". If you use chrootedfs, you should just pass relative path by cutting chrooteduri. Since we support only at root, it will be simply "/" on chorootedfs.

I have one another concern that we are returning absolute path with fall back in ls. But with regular mountlinks we will return relative to viewfs scheme ex: [viewfs://default/data, viewfs://default/targetRoot, viewfs://default/danglingLink, [viewfs://default/user2, viewfs://default/user, viewfs://default/internalDir, viewfs://default/linkToAFile]
Since fallback also part of viewfs and its actually a child fs, should we return fallback paths also relatively constructed with respective to viewfs:// scheme?
exmaple: your fallback path is hdfs://nn1/fallBackDir and you have a dir under it. That is /user1
So, when you do ls, should it return viewfs://default/user1 ?
if you use this ls result path to create some directory like viewfs://default/user1/test1, it will/should work ( IMO) and it should create at physical location of hdfs://nn1/fallBackDir/user1/test1
Because you did not have any other mount link with /user1, it will fallback to create at fallback link.

But with the current behavior, we are returning hdfs://nn1/fallBackDir/user1 and if you use this path to create a dir under it, it will actually go to regular hdfs instance but not via viewfs. This could be a potentially problem in that case overloadScheme impl. There hdfs can go to ViewFSOverloadScheme and try to create path as hdfs://nn1/fallBackDir/user1. Now fallBackDir/user1 will be attempted to create in viewfsOS env, it checks whether we have any link with fallBackDir. Of course we will not have any regular link. So, it will create at fallback link.
Our fallback link is already hdfs://nn1/fallbackDir and now it will attempt to create /fallbackDir/user1
So, it will result to create hdfs://nn1/fallbackDir/fallbackDir/user1. This seems not correct.

FileStatus[] mountPointStatuses) {
ArrayList<FileStatus> result = new ArrayList<>();
Set<String> pathSet = new HashSet<>();
int i = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove above unused variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 31s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 51s Maven dependency ordering for branch
-1 ❌ mvninstall 20m 37s root in trunk failed.
+1 💚 compile 17m 43s trunk passed
+1 💚 checkstyle 2m 44s trunk passed
+1 💚 mvnsite 3m 3s trunk passed
-1 ❌ shadedclient 20m 20s branch has errors when building and testing our client artifacts.
+1 💚 javadoc 1m 55s trunk passed
+0 🆗 spotbugs 3m 10s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 22s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 2m 1s the patch passed
+1 💚 compile 17m 24s the patch passed
+1 💚 javac 17m 24s the patch passed
-0 ⚠️ checkstyle 2m 44s root: The patch generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105)
+1 💚 mvnsite 2m 53s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
-1 ❌ shadedclient 15m 11s patch has errors when building and testing our client artifacts.
+1 💚 javadoc 1m 51s the patch passed
+1 💚 findbugs 5m 31s the patch passed
_ Other Tests _
+1 💚 unit 9m 30s hadoop-common in the patch passed.
-1 ❌ unit 100m 9s hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
230m 12s
Reason Tests
Failed junit tests hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks
hadoop.hdfs.TestFileChecksumCompositeCrc
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux ce30e55b40eb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ac4a2e1
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/artifact/out/branch-mvninstall-root.txt
checkstyle https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/artifact/out/diff-checkstyle-root.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/testReport/
Max. process+thread count 5089 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/7/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@umamaheswararao
Copy link
Contributor

Could you also check CI results? Seems like checkstyle issue related to this change? Thank you.

@umamaheswararao
Copy link
Contributor

Thank you @abhishekdas99 for addressing all the comments. Nice work!
+1 on latest changes and pending CI results. Thanks

@abhishekdas99
Copy link
Contributor Author

Thank you @abhishekdas99 for addressing all the comments. Nice work!
+1 on latest changes and pending CI results. Thanks

Thanks @umamaheswararao for your guidance.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 24m 36s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 0m 51s Maven dependency ordering for branch
+1 💚 mvninstall 23m 15s trunk passed
+1 💚 compile 20m 46s trunk passed
+1 💚 checkstyle 3m 2s trunk passed
+1 💚 mvnsite 2m 57s trunk passed
+1 💚 shadedclient 22m 11s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 1m 52s trunk passed
+0 🆗 spotbugs 3m 27s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 5m 46s trunk passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 2m 13s the patch passed
+1 💚 compile 20m 31s the patch passed
+1 💚 javac 20m 31s the patch passed
+1 💚 checkstyle 2m 59s the patch passed
+1 💚 mvnsite 3m 0s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 15m 17s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 2m 0s the patch passed
+1 💚 findbugs 6m 6s the patch passed
_ Other Tests _
+1 💚 unit 10m 11s hadoop-common in the patch passed.
-1 ❌ unit 122m 37s hadoop-hdfs in the patch passed.
-1 ❌ asflicense 1m 5s The patch generated 10 ASF License warnings.
289m 30s
Reason Tests
Failed junit tests hadoop.hdfs.TestEncryptionZonesWithKMS
hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
hadoop.hdfs.TestRollingUpgradeRollback
hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
hadoop.hdfs.TestReconstructStripedFile
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
hadoop.hdfs.TestFileChecksum
hadoop.hdfs.TestErasureCodingPolicies
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
hadoop.hdfs.TestWriteReadStripedFile
hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId
hadoop.hdfs.TestDFSInputStreamBlockLocations
hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped
hadoop.hdfs.TestTrashWithEncryptionZones
hadoop.hdfs.TestDataTransferKeepalive
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/8/artifact/out/Dockerfile
GITHUB PR #2010
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux f45ad67e26fa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 7bb902b
Default Java Private Build-1.8.0_252-8u252-b09-1~18.04-b09
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/8/testReport/
asflicense https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/8/artifact/out/patch-asflicense-problems.txt
Max. process+thread count 3258 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-2010/8/console
versions git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@umamaheswararao umamaheswararao merged commit ce4ec74 into apache:trunk May 19, 2020
@umamaheswararao
Copy link
Contributor

Thank you @abhishekdas99 for the contribution. I have just committed this to trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants